5/19/2021

Introduction

  • Global pandemic
  • Transition to online instruction
  • How can we characterize this transition?

A Quasi-Experiment

  • Louisiana Tech is on a quarter schedule with semester hours.
  • SP20 had only begun when we transitioned online.

A Quasi-Experiment

A Hypothesis

We expected that the grade distribution would be increasingly bi-modal, with students either excelling or failing under the circumstances. We presumed this outcome because many of the “A” students will be “A” students no matter the circumstances. Some students have the intellect, but lack the discipline to complete a course online. We expected those students to complete in-person coursework with “Bs” and “Cs,” and we were not certain that those students would complete the online quarter.

Holderieath, J.J., M.K. Crosby, T.E. McConnell, and D.P. Jackson. 2021. “Impact of COVID-19-Related Transition to Online Instruction on Student Achievement.” Applied Economics Teaching Resources (AETR) 3(1).

Initial Graphical Analysis

“Grades Awarded During Spring Term from 2018 to 2020 at Louisiana Tech University School of Agricultural Sciences and Forestry”

Holderieath, J.J., M.K. Crosby, T.E. McConnell, and D.P. Jackson. 2021. “Impact of COVID-19-Related Transition to Online Instruction on Student Achievement.” Applied Economics Teaching Resources (AETR) 3(1).

Poisson Regression

  • Compared the two previous spring terms to SP 21, using SASF data.
  • Compared Proportion earning As, Proportion Finishing, and Proportion Passing
  • No significant relationship between the SP21 dummy and the dependent variables.
  • More detailed graphical analysis showed that there was not a single common experience.
  • Some classes did not see that much change.
  • Some classes, especially difficult classes, had much higher drop rates. A few classes had fewer students drop!
  • Some classes had much higher rates of students earning A’s and some classes had much lower rates.

Holderieath, J.J., M.K. Crosby, T.E. McConnell, and D.P. Jackson. 2021. “Impact of COVID-19-Related Transition to Online Instruction on Student Achievement.” Applied Economics Teaching Resources (AETR) 3(1).

Marginal Effects Via Prediction

Holderieath, J.J., M.K. Crosby, T.E. McConnell, and D.P. Jackson. 2021. “Impact of COVID-19-Related Transition to Online Instruction on Student Achievement.” Applied Economics Teaching Resources (AETR) 3(1).

Historical Comparisons

Holderieath, J.J., M.K. Crosby, T.E. McConnell, and D.P. Jackson. 2021. “Impact of COVID-19-Related Transition to Online Instruction on Student Achievement.” Applied Economics Teaching Resources (AETR) 3(1).

History By Class

Finishing

Holderieath, J.J., M.K. Crosby, T.E. McConnell, and D.P. Jackson. 2021. “Impact of COVID-19-Related Transition to Online Instruction on Student Achievement.” Applied Economics Teaching Resources (AETR) 3(1).

History By Class

Passing

Holderieath, J.J., M.K. Crosby, T.E. McConnell, and D.P. Jackson. 2021. “Impact of COVID-19-Related Transition to Online Instruction on Student Achievement.” Applied Economics Teaching Resources (AETR) 3(1).

History By Class

Earning an A

Holderieath, J.J., M.K. Crosby, T.E. McConnell, and D.P. Jackson. 2021. “Impact of COVID-19-Related Transition to Online Instruction on Student Achievement.” Applied Economics Teaching Resources (AETR) 3(1).

The Pandemic Continues

  • Louisiana Tech continued with a mix of online and reduced capacity in-seat instruction.
  • Added pressure of the political environment of the country.
  • Zoom fatigue

More Pandemic, More Data

  • We requested grade data for the entire University covering each quarter since SP17 (term 20183)
  • 65,509 individual grades with course, section, instructor, and term.
  • Delivery method can be inferred with some error this year from section.

A Simple Story

Just kidding.

“Percent Earning an A By Department and Term”

University Data

Pass, Fail, and Withdraw

University Data

Percent Awarded For Each Letter Grade By Quarter

University Data

SP20 Median Almost 60% A’s

University Data Poisson

The Data

65,509 Individual grades, by Course, Instructor, and Section
Instructor Term Grade ID FULL_ID Dept Course_Number Section Honors Online Upper Grad pfw term_label quarter
086 : 4755 20191 : 7722 A :29778 HIST202: 2903 Length:65509 MATH : 7681 202 : 4960 001 :29300 0:64253 0:60517 0:41986 0:60594 PASS:54546 FALL 18 : 7722 FALL :22322
038 : 2291 20211 : 7653 B :14932 HIST201: 1467 Class :character HIST : 6124 100 : 3912 002 : 7207 1: 1256 1: 4992 1:23523 1: 4915 FAIL: 5088 FALL 20 : 7653 WINTER:19125
028 : 1983 20201 : 6947 C : 8160 FYE100 : 1393 Mode :character CHEM : 5229 201 : 3756 V84 : 4992 NA NA NA NA W : 4777 FALL 19 : 6947 SPRING:19450
155 : 1896 20193 : 6782 W : 4777 HIST101: 1114 NA KINE : 4178 101 : 3376 003 : 3439 NA NA NA NA IC : 5 SPRING 19 : 6782 SUMMER: 4612
041 : 1827 20183 : 6626 D : 2564 MATH240: 1095 NA CSC : 2583 103 : 2094 051 : 1894 NA NA NA NA NA’s: 1093 SPRING 18 : 6626 NA
070 : 1714 20192 : 6565 (Other): 4205 CHEM101: 1052 NA (Other):39674 290 : 1735 004 : 1462 NA NA NA NA NA WINTER 18-19: 6565 NA
(Other):51043 (Other):23214 NA’s : 1093 (Other):56485 NA NA’s : 40 (Other):45676 (Other):17215 NA NA NA NA NA (Other) :23214 NA

University Data Poisson

Passing

University Data Poisson

Passing

term estimate std.error statistic p.value
Online1 -0.0346522 0.0129965 -2.6662829 0.0076695
Honors1 0.0377259 0.0220513 1.7108257 0.0871133
Upper1 0.1936615 0.2325181 0.8328878 0.4049080
Grad0 -0.3055180 0.1872973 -1.6311924 0.1028497
term_label_SPRING.20 0.0237425 0.0110814 2.1425579 0.0321486
term_label_SUMMER.20 0.0217746 0.0172590 1.2616346 0.2070803
term_label_FALL.20 0.0151580 0.0124922 1.2133980 0.2249777
term_label_WINTER.20.21 0.0256013 0.0125247 2.0440650 0.0409471

University Data Poisson

Finishing

University Data Poisson

Finishing

term estimate std.error statistic p.value
Online1 -0.0317029 0.0129929 -2.4400211 0.0146864
Honors1 0.0212325 0.0223573 0.9496888 0.3422704
Upper1 0.1840394 0.1265453 1.4543358 0.1458532
Grad0 -0.3545206 0.2350633 -1.5081922 0.1315053
term_label_SPRING.20 -0.0271166 0.0111676 -2.4281455 0.0151763
term_label_SUMMER.20 0.0484518 0.0178050 2.7212439 0.0065037
term_label_FALL.20 0.0067780 0.0123398 0.5492765 0.5828157
term_label_WINTER.20.21 0.0301389 0.0122414 2.4620505 0.0138145

University Data Poisson

Earning an A

University Data

Earning an A

term estimate std.error statistic p.value
Online1 0.0488437 0.0166566 2.9323881 0.0033637
Honors1 0.1365372 0.0222990 6.1230047 0.0000000
Upper1 0.5350573 0.3994002 1.3396521 0.1803585
Grad0 -0.9576176 0.4020000 -2.3821334 0.0172127
term_label_SPRING.20 0.0663576 0.0140751 4.7145395 0.0000024
term_label_SUMMER.20 0.0126206 0.0235299 0.5363636 0.5917073
term_label_FALL.20 0.0220989 0.0161802 1.3657940 0.1720036
term_label_WINTER.20.21 0.0245207 0.0157408 1.5577730 0.1192871

Where to from here?

Technical Issues

  • Robust Standard Errors
  • We are missing major predictors of grades – the attributes of students!
  • Probably should use something other than Poisson.

Where to from here?

Conceptual Issues

  • How to look at change over time?
  • I do not believe this to be appropriately modeled as panel data. The students are different, so the class is different.
  • But there is a meaningful similarity between a class in one term and the next.
  • Arguably grade outcomes in one term have something to do with grade outcomes in the next (with same instructor and class).
  • 2 and 3 step estimator to sort course/section/instructors into types
  • At some point (I think I am there) a parametric approach loses its appeal.

Thank You!

  • Are there any questions?